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(57) ABSTRACT 

A system and method for providing user input to an appli- 
cation executing on an interactive television system wherein 
a user provides non-textual information to the interactive 
television system and this information is converted by a 
server to textual information which is provided to the 
application. In one embodiment, a digitizer pad coupled to 
a set-top box is used to digitize the user's handwriting. The 
digitized information is conveyed to a remote server which 
converts the digitized handwriting data into textual infor- 
mation. The textual information is conveyed to the set -lop 
box, where it is input to an application executing on the 
set-top box. In another embodiment, a microphone is 
coupled to a set-top box. The microphone allows the user to 
input voice information which is digitized and conveyed to 
the server for conversion into textual information. The 
textual information is conveyed back to the set-top box and 
is input to an application executing on the set-top box. 

22 Claims, 3 Drawing Sheets 
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INTERACTIVE TELEVISION SYSTEM AND choice from a menu. The application may, however, also 

METHOD FOR CONVERTING NON- require the user's name or shipping information which 

TEXTUAL INFORMATION TO TEXTUAL cannot simply be selected from a menu. Another example of 

INFORMATION BY A REMOTE SERVER an application which requires textual information is a mes- 

Tw™^r. rt ,T^ ^ ^, ¥T , „„„^™„ T 5 sa ging application for which the viewer must provide the 

BACKGROUND OF THE INVENTION message^ well as information identifying the addressee of 

1. Field of the Invention the message. Some means is therefore necessary to enter this 
The invention relates generally to interactive television textual information. 

systems and more particularly to means and methods for One option for entering text would be to use a keyboard, 

using a server to convert user-provided information into a 10 This might not be the best solution, however, because some 

data format which can be used by an interactive television users may not feel comfortable using a keyboard. One of the 

application. attractions of interactive television systems is the ease with 

2. Description of the Relevant Art which they may be used. Interactive television applications 
Interactive television systems can be used to provide a are dcsi g ncd to provide simplified user interfaces and many 

variety of services to users. These systems are capable of K< * mrt 00 more *P ut than can be provided using a basic 

displaying text and graphic images to facilitate viewer remole COttl ™ 1 ^ requirement of entering textual infor- 

interaction as well as the audio and video streams associated ma,lon ™ 3 ke y board ma y make the interactive television 

with ordinary television programs. Interactive television P!f m less attractive to users because of the additional 

systems enable viewer interaction and thereby allow the , n hardware wh "* * rw J uircd fo [ »"e system, the perceived 

systems to be used for marketing and educational purposes 2 ° «>mP!«ity of the system or the difficulty which may be 

in addition to extending the entertainment capabilities of an "Ponced by some users m typing the necessary informa- 

ordinary television. Viewers can order advertised products Further ' m ^ 1«W»S* (e-g. Chinese,) the com- 

or services, request information regarding particular J* 1 ]? wntten language makes text entry via a 

programs, or send electronic messages (e-mail). ke y board dlfficul1 even for experienced users. 

In a typical interactive television system, a broadcast °! her for P rovidin S ^al information to an 

service provider generates an interactive television signal for f Pfhcation may involve means for converting non-textual 

transmission to a viewer's television. The interactive tele- ^formation mto a textual form. For example, using voice 

vision signal includes an audio-video portion consisting of a recognition technology, a user's voice can be sampled and 

television program, as well as an interactive portion con- 30 compared to previously sampled speech patterns to deter- 

sisting of application code or control information. The muic thc w ° rds s P° ken b * * c . ^ words can &™ 

broadcast service provider combines the audio-video and ? ut P ut b y the ««)gmtion system as text. Handwrit- 

interactivc portions into a single signal for transmission to a m S recognition systems could also be used to generate 

receiver connected to the user's television. The signal is textual infor mation for the application. These systems per- 

typically compressed prior to transmission and transmitted 35 form lhe P rocess OD una £ es of me user ' s handwriting 

through broadcast channels such as cable television (CATV) ( entercd via a Panics tablet or similar input device) to 

lines or direct satellite transmission systems. determine the text wntten by the user. While these technolo- 

Ttae interactive functionality of the interactive television gies have evolved to the poiiit that they can reliably generate 

system is controlled by a set-top box connected to the accurate textual information from the users voice or 

television. The set-top box receives the signal transmitted by 40 dwn m S' * ? "ST" ^T*' 

the broadcast service provider, separafes the interactive ™ e a PP hcalIons ^ * verv large and they need large 

portion from the audio-video portion and decompresses the am0l ! nls of Pressing tune to perform the required pattern 

respective portions of the signal. If the interactive portion of m ^ 11,656 technoIo f*s therefore cannot be .mple- 

the signal comprises an interactive application, the applica- mented «" currcnl s .et-top boxes, wh.ch have very limited 

lion may be executed while the audio-video information (the 45 St0fagC and P"** 551 ^ ca P acil y- 

normal television signal) is conveyed to the television. The SUMMARY OF THE INVENTION 
set-top box may combine the audio-video information with 

interactive graphics or audio generated by the interactive One or more of the problems outlined above may be 

application prior to conveying the information to the tele- solved by the various embodiments of the invention. The 

vision. The interactive graphics and audio may present 50 invention comprises a system and method for enabling a 

additional information to the viewer or may prompt the user to provide non-textual information which is converted 

viewer for input and may be designed to function in response by the system to a textual form in which it can be used by 

to signals in the broadcast or actions taken by the viewer. An the interactive application. The non-textual information is 

interactive television system may also run applications that entered by the user at the set- top box of a receiving station 

present the user only with the audio and graphics relating to 55 and this information is transmitted to a server which may be 

the interactive application. In other words, this information located at a broadcast station. The server converts the 

may be displayed in place of the normal television signal, information into textual data so that it can be used by the 

rather than adding to the signal. system. In one embodiment, the server transmits the textual 

There are various examples of interactive applications data back to the receiving station, where it can be used by 

which require viewer input. One example is a televised 60 an application executing in the set-top box. In other 

buying service in which a series of products are displayed embodiments, the textual data can be used at the server or 

via a television program and the viewer utilizes the inter- transmitted to a part of the system other than the set-top box. 

active functions of the television system to purchase the One embodiment comprises an interactive television sys- 

displayed products. The viewer must identify the product he tem comprising a broadcast station and a receiving station, 

or she wishes to purchase and may also need to provide size, 65 The broadcast station transmits an interactive television 

color, and other information descriptive of the product. This application to the receiving station, which then executes the 

information may be provided by selecting the appropriate application. The application requires textual data from the 
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user. The receiving station includes a set-top box which 
executes the application and a graphics tablet which is 
coupled to the set-top box for entering information. The user 
provides Ibe information by writing on the graphics tablet, 
which generates an image file. The image file is transmitted 
from the set-top box to the broadcast station. The broadcast 
station includes a server which uses handwriting recognition 
software to convert the image file into character or textual 
data. The textual data is transmitted back to the application 
at the set-top box, which uses the data as if it had been typed 
in directly by the user. 

Because the handwriting recognition software resides on 
the server computer instead of the set-top box, more 
resources are available for execution of the software. The 
server typically has more available memory and more pro- 
cessing power than the set -top box and consequently pro- 
vides much faster recognition of handwriting images. The 
greater available resources enable the software to provide 
for recognition of additional, complex languages (e.g., 
Chinese) and extended character sets (e.g., Unicode). Since 
the recognition software is maintained on the server, the 
software can be quickly and easily updated with the latest 
handwriting recognition technology, and it does not have to 
be distributed to the individual subscribers' set-top boxes. 

In one embodiment, the textual data need not be sent back 25 
to the set-top box, but may instead be used at the server or 
another location remote from the set-top box. For example, 
a user may place an order for a product by entering oral or 
handwritten information and the order may be confirmed 
later by e-mail or other means. In another embodiment, the 
server could consist of a human operator who receives the 
audio or image data and manually transcribes it into textual 
information. The audio or image data may alternately be 
converted into textual information by automated means 
operating under human supervision. 

In an alternate embodiment, the interactive television 
system includes a microphone rather than a graphics tablet 
for entry of information to the system. The microphone is 
used to provide voice data, which is recorded and transmit- 
ted to a server equipped with a voice recognition application. 
The voice recognition application converts the voice data 
into textual data, which is then transmitted back to the 
application executing on the set-top box. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Other objects and advantages of the invention will 
become apparent upon reading the following detailed 
description and upon reference to the accompanying draw- 
ings in which: 

FIG. 1 is a block diagram illustrating one embodiment of 
a system for distribution of interactive television applica- 
tions and television programs from their sources to a series 
of viewers. 

FIG. 2 is a block diagram of a set-top box in one 
embodiment of the invention. 

FIG. 3 is a block diagram illustrating the components of 
a graphics tablet in one embodiment of the invention. 

FIG. 4 is a flow diagram illustrating the flow of 
handwritten/textual information in one embodiment of the 
invention. 

FIG. 5 is a flow diagram illustrating the manner in which 
the server's handwriting recognition application processes 
the image data to produce textual data in one embodiment of 
the invention. 

While the invention is susceptible to various modifica- 
tions and alternative forms, specific embodiments thereof 
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are shown by way of example in the drawings and will 
herein be described in detail. It should be understood, 
however, that the drawing and detailed description thereto 
are not intended to limit the invention to the particular form 
disclosed, but on the contrary, the intention is to cover all 
modifications, equivalents and alternatives falling within the 
spirit and scope of the present invention as defined by the 
appended claims. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 

One embodiment of the inventive method is described in 
detail below. In this embodiment, an interactive television 
system transmits an audio-video-interactive signal from a 
broadcast station to a receiving station via a broadcast 
channel. ("Broadcast" is used herein to refer to transmission 
of a single signal to all subscribing receivers.) The broadcast 
channel may comprise a direct satellite transmission channel 
or any known means for broadcasting a signal, including 
non-satellite, cable, telco, MMDS (microwave) and terres- 
trial transmissions. (A "direct" satellite transmission as used 
herein means a transmission received by the interactive 
television receiver directly from the satellite.). The receiving 
station in this embodiment is additionally configured to 
receive signals via a modem connection to the broadcast 
station. 

The audio-video-interactive signal broadcast to the 
receiving station may contain both television programming 
and interactive information such as control signals or inter- 
active applications. When the broadcast signal is received, it 
is separated into its components and processed (e.g., 
decompressed) to reconstruct the respective television pro- 
gramming and interactive signals. An interactive application 
which was broadcast to the receiving station or which was 
resident in the receiving station is executed on the micro- 
processor of the set-top box. The application is one which 
requires textual data from the user. The user, however, is 
allowed to input the data in a non-textual form. "Non- 
textual" as used herein means a form handled by the system 
as something other than a series of alphanumeric characters, 
such as the ASCII character set. A graphical representation 
of handwriting, although textual in nature, is handled as an 
image or as vectorized handwriting information and not as 
the characters or words which are written. "Textual," on the 
other hand, means one or more characters or words. If the 
system is set up for an English-speaking user, the characters 
may be from the ASCII character set. If the system is set up 
for a non- English-speaking user, the characters will be from 
the character set of the user's language (e.g., Kanji charac- 
ters for a Japanese user). 

The set -top box of the receiving station is configured with 
a device for non-textual data entry. In one embodiment, this 
device is a graphics tablet on which the user can inscribe 
(write) the information required by the application. The 
graphics tablet digitizes the user's handwriting and stores 
the digitized data as an image file. The set-top box transmits 
the file containing the digitized data to the broadcast station 
for conversion into a textual form. (The image data need not 
all be contained in a single file, and may be transmitted to 
the server in segments to spread out the processing load.) 
The broadcast station includes a server which is configured 
to transform image data into textual data using handwriting 
recognition algorithms. The broadcast station then transmits 
the textual data back to the receiving station. The application 
executing on the set-top box identifies the received textual 
data as corresponding to the image data which it earlier 
transmitted to the broadcast station. The set-top box accord- 
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ingly utilizes the textual data as if it had been entered as text Id addition to the broadcast channel between the broad- 
by the user of the set-top box and continues execution of the cast station and receiving station, there may be other 

interactive television application. channels, such as a modem channel (which may also be 
Referring to FIG. 1, a block diagram illustrating a system referred to as an http channel, or hypertext transfer protocol 
for distribution of interactive television applications and 5 channel.) These types of channels serve two functions in the 
television programs from their sources to a series of viewers system: they allow the set-top box to provide data to the 

is shown. Broadcast station 10 has a television program broadcast station; and they provide an alternate path for 

source 11 and an interactive application source 12. The from sources U-13 to be delivered to receiving station 20. 

television program source may include remote broadcast It should also be noted that, if receiving station 20 and server 

network feeds, videotape recorders, computers, data storage w 13 are connected via a transmission medium other than the 

devices, and the like. Interactive application sources 12 may broadcast station's broadcast channel and return path, the 

provide interactive applications, control information or xrvcT & not constrained to be located at the broadcast 

audio or video information which is to be included in the station . It may ^ more to locatc 13 at a 

mteractive television signal. Additionally, broadcast station sitc whicQ b te from broadcast ^ 10 m ^ 

10 includes a server 13, which is used to process no n- textual reduce mc workload of tbc broadcas( staU 

data received at the broadcast station and generate textual 15 ■ ,, M . . . . . »«"""f *« 

data. Tne information generated by the television program cfficienlly 5°°^ ^ bctween ^ receivm S station and the 

source 11, interactive application source 12 and serverl3 is ^ 0r for other reasons " 

typically processed by compression/packelization units . Refcrn og to FIG. 2, a block diagram of a set-top box 22 

14-16 before it is broadcast. Usually, the information is also °. ne cmbcxliment is shown. The broadcast signal is 

compressed in order to conserve bandwidth. (It should be 20 received and fed into tuner 31. Tuner 31 selects the channel 

noted that while this embodiment comprises a digital broad- 0D which the broadcast audio-video-interactive signal is 

cast system, other embodiments may comprise analog transmitted and passes the signal to processing unit 32. 

broadcast systems in conjunction with means for transmit- (Tuner 31 may be replaced by other means, all collectively 

ting digital application data. The analog broadcast systems referred to herein as input ports, for receiving signals from 

may include existing television networks, and the means for 25 various signal sources.) Processing unit 32 demultiplexes 

transmitting the application data may include transmissions the packets from the broadcast signal if necessary and 

in the vertical blanking interval or parallel transmissions via reconstructs the television programs and/or interactive 

telephone lines or out-of-band cable signals.) applications embodied in the signal. The programs and 

Any of a number of compression algorithms, such as one applications are then decompressed by decompression unit 

of the Motion Picture Expert Group (MPEG) compression 30 33. The audio and video information associated with the 

standards, may be used if appropriate for a particular pro- television programs embodied in the signal is then conveyed 

gram or application. The information is packetized to enable to display unit 34, which may perform further processing 

error checking, interleaving of data and other transmission- and conversion of the information into a suitable television 

related functions. Additional data may accordingly be format, such as NTSC or HDTV audio/video. Applications 

appended to the application and programming data. For 35 reconstructed from the broadcast signal are routed to random 

example, error check sums may be appended for error access memory (RAM) 37 and are executed by micropro- 

detection/correction and time stamps may be included for cessor 35. Graphics tablet 39 provides a means for the user 

the purpose of synchronizing associated audio and video to supply handwritten information for conversion to text and 

signals. The packetized information from compression/ subsequent use by the applications. (In other embodiments, 

packetization units 14-16 is fed into multiplexing unit 17, 40 graphics tablet 39 may be replaced by a microphone for 

which intersperses the packets prior to transmission. The supplying voice data or some other type of input device for 

interspersed packets are then broadcast to the receiving supplying non- textual information to the system.) 

stations 20. (Although only one receiving station is shown in Microprocessor 35 may comprise various types of 

the figure, it is contemplated that the audio-video-interactive microprocessors, microcontrollers, digital signal processors 

signal is broadcast to a group of subscribing receiving 45 (DSPs), or other types of software instruction processing 

stations.) In the figure, the audio-video-interactive signal is devices, as are appropriate to the particular design. RAM 37 

depicted as being transmitted via satellite broadcast through may include memory units which are static (e.g., SRAM), 

antenna 19. dynamic (e.g., DRAM), volatile or non-volatile (e.g., Flash 

The broadcast signal is relayed by communications sat- memory), as required to support the functions of the set-top 

ellite 28 and received by receiving station 20. Although the so box. When power is applied to the set-top box, micropro- 

figure illustrates a satellite transmission, it is contemplated cessor 35 executes operating system code which is stored in 

that any broadcast medium (e.g., CATV or direct satellite ROM 36. (In some embodiments, ROM 36 may comprise 

transmission) may be used. Receiving station 20 is contem- Flash memory or EEPROMs.) The operating system code 

plated to be one of a number of such stations which are executes continuously while the set-top box is powered in 

subscribers of the broadcast service provider operating 55 the same manner the operating system code of a typical 

broadcast station 10. The broadcast signal is collected by personal computer (PC) and enables the set-top box to act on 

receiving antenna 21 and fed to set-top box 22. Set-top box control information and execute interactive and other appli- 

22 processes the packetized signal to reconstruct the televi- cations. The set-top box also includes modem 38. Modem 38 

sion programs and interactive applications embodied in the provides both a return path by which viewer data can be 

signal. The reconstructed applications are executed in the 60 transmitted to the broadcast station and an alternate path by 

set-top box, while the reconstructed television programs are which the broadcast station can transmit data to the set-top 

passed to television 23, where they are displayed. (It is noted box. 

that television 23 can be any suitable monitor or display Although the term "set-top box" is used herein, it is 

device.) The interactive applications may generate graphics understood that this term refers to any receiver or processing 

or audio which are combined with the television program 65 unit for receiving and processing a transmitted signal and 

prior to being displayed. The interactive applications may conveying the processed signal to a television or other 

also be displayed in place of the television programming. monitor. The set-top box may be in a housing which 
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physically sits on top of a television, it may be in some other microprocessor 35, it is contemplated that a single device 

location external to the television (e.g., on the side or back may be sufficient to perform both functions, 

of the television or remotely located from the television), or Digitizer 52 may also be constructed using one of a 

it may be incorporated into the television itself. number of technologies. Although early technologies often 

Alternatively, the functionality of the set-top box may be 5 used opaque structures, technologies used in modem digi- 

entirely removed from the television and placed in a more tizers are more likely to incorporate transparent panels 

remote location, such as outside a house in which the scttop which are designed to be used in conjunction with overlaid 

box is located. Set-top box 22 serves to demodulate (if displays. Digitizer 52 may use electrostatic, resistive film or 

necessary) the signal received from broadcast station 10 and capacitive/electrostatic film technologies. Various embodi- 

to separate the components of the signal, such as different 10 ments of the invention may incorporate different ones of the 

television programs and interactive applications. Other technologies which are best suited to the particular embodi- 

embodiments of the set-top box may have different compo- ments. Digitizer 52 senses the position of the digitizer pen 

nents or interconnections than those shown in FIG. 2. and transmits the corresponding X-Y coordinates to the 

Similarly, television 23 may be a television or a video set-top box. In one embodiment, this X-Y data is recorded 

monitor employing any suitable television format (e.g., 15 a XTic5 of darkened pixels in a bitmap image of the text 

NTSC or HDTV), or it may be replaced by other devices, wriMeo b ? ^ ^ ^ ima S e « tben transmitted to the 

such as a video recorder, depending on the particular " rver for Pressing and recognition of the written data, 

embodiment. recognition of the bitmap image is sometimes referred 

Referring to FIG. 3, a block diagram illustrating the to as off line recognition. 

«r „ „,„„u;™. ,.Kt». en „i ^. _ ■ In an alternate embodiment, digitizer 52 may sense not 

t « / g £ ?<n V . <7h-, « 20 onl y XY P° sili0Q . but »1» the direction of the pen's path, 

components of graphics tablet 50 are display 51, digitizer 52 the ' , s ^ d fln(j acceleralio0i and olhe 7 inf 

and processor 53. Graphics tablet 50 can have a wide range lion which £ rms F a vectorized reprcaenIalion of , he user's 
of sizes, from several inches across (as used in pen com- handwriting. This vectorized information can be transmitted 
putmg systems) to several feet across (as used in some CAD t o the server as it is produced so that recognition can be 
systems). It is contemplated that the most convenient size for 2 $ performed as the user is writing. The use of vectorized 
the graphics tablet will be the size of a small or medium information to recognize handwritten information is some- 
sized notepad. The surface of graphics tablet 50 should be times referred to as on-line recognition. 
Bat and smooth to facilitate use as a writing surface. The receiving station is operatively connected to the 
Although graphics tablet 50 is contemplated to be connected broadcast station by a broadcast channel. This broadcast 
to set-top box 22 by a data cable, infrared transmission or 30 channel can utilize various transmission media and is con- 
any other suitable transmission means may also be used to templated to include media such as coaxial cable and free 
transmit data between graphics tablet 50 and set -top box 22. space (e.g., as used for direct satellite transmissions.) The 
To the extent that graphics tablet 50 is not constrained by broadcast channel forms a transmission path between the 
these data transmission means, the user should be able to broadcast station and the receiving station. The broadcast 
move the tablet to a convenient and comfortable writing 35 station and receiving station are also connected by a return 
position. path. The return path typically consists of a pair of modems, 

Display 51 and digitizer 52 are overlaid so that the path one in the receiving station and one in the broadcast station, 

of a digitizer pen (not shown) can be displayed as the user each connected to a standard telephone line. Other means for 

writes on the tablet, thereby simulating writing on a notepad. establishing a return path (e.g., using a portion of the 

Display 51 is contemplated to be a flat panel display using 40 bandwidth of the broadcast channel) are also contemplated. 

LCD (liquid crystal diode) or similar technologies. These The system illustrated in FIG. 1 utilizes the broadcast 

technologies use substances this those between two plates of channel to establish a return path for communicating data 

glass to control the amount of light which passes through the from the receiving station to the server, 

display. Depending upon the chosen technology, the display In one embodiment, an application is transmitted from the 

may simply control the amount of light from any source 45 broadcast station to the receiving station, where it is recon- 

behind the display, or it may generate varying amounts of structed and executed. In alternate embodiments, the appli- 

light from each pixel in the display. The display can be cation may be resident in the set-top box or may be provided 

configured to provide feedback to the user by displaying his to the set-top box by means other than the broadcast channel 

or her handwriting, or it may display icons or other images (e.g., flash card.) FIG. 4 is a flow diagram illustrating the 

representative of the user's input. The display may. also be 50 flow of handwritten/textual information in the system. While 

configured to display forms, or other prompts, in response to the application is executing 6 1 , it prompts the user for some 

which the user can provide handwritten information. In an sort of textual information 62. The user enters the informa- 

alternate embodiment, graphics tablet 50 may be simplified lion on the graphics tablet 63, writing the information in his 

by eliminating display 51. In such an embodiment, the user or her normal handwriting. The graphics tablet digitizes the 

could instead receive visual feedback, including prompts, 55 image of the user's writing and conveys the digitized data to 

forms, handwriting and other images through television 23. the set-lop box, which transmits the information to the server 

The combination of display 51 and digitizer 52 should be 64 via the return path to the broadcast station. The image 

selected to provide sufficient resolution to allow an accurate data is received by the broadcast station and passed on to the 

digital representation of the user's handwriting. It is con- server. The server processes the image data 65, recognizing 

templated that the resolution of the digitizer and display 60 characters and/or words in the image and producing the 

should be roughly equivalent to that of a fairly high-quality equivalent textual data. This textual data is then transmitted 

printer. The combination should also provide display quality back to the receiving station 66, either via the broadcast 

(e.g., brightness and contrast) which makes the handwritten channel or via the return channel. The textual data is then 

information easy for the user to see. Microprocessor 53 utilized by the application executing in the set-top box 67 as 

should be fast enough that the path of the digitizer pen is 65 if the text had been directly entered by the user, 

displayed without any substantial delay as it is drawn by the FIG. 5 is a flow diagram illustrating the manner in which 

user. Although microprocessor 53 is shown separately from the server's handwriting recognition application processes 
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the image data to produce textual data. An off-line, bitmap may segment the image data and then convert the data to 

recognition system is used in this embodiment. It is assumed text, or it may convert the entire image to text and then parse 

in this instance that the user's handwriting is not contained the text to determine the recipient *s address. In an alternate 

in a form, but is instead free-form input to the graphics embodiment, only the address of the message is converted to 

tablet. Further, in one embodiment, the handwriting image is 5 text while the body of the message is transmitted to the 

transmitted to the server in segments as they are generated recipient as an image. Although the address of the e-mail 

by the graphics tablet. This may serve both to reduce the must be computer-readable so it can be properly routed to 

peak loading of the server resulting from recognition of the the recipient, the body of the message need not be converted 

image and to allow the server to provide simultaneous to text — the recipient can read the message embodied in the 

feedback (i.e., recognized characters) to the user. 10 image whether or not it is converted to text data. In fact, it 

("Simultaneous" as used here means that some recognized may in some cases be preferable to deliver some messages 

text is transmitted to the set-top box as the user continues to as image data so that the sender can communicate drawings 

write on the graphics tablet so that the user can see the or other non-textual data. Delivery of the image of a 

results of the handwriting recognition.) If the image instead handwritten note may also be preferable because simple text 

combines text with a known form, the form is first identified 15 messages may be considered impersonal, 

and removed. Then, if the handwriting was confined to The user may be given the option of selecting an 

several boxes or fields on the form, these fields are isolated addressee from an address book or a list of previous address- 

and handled individually. (Free-form handwritten entries ees. A menu can be presented to the user via the graphics 

may be distinguished from each other in the same manner as tablet to allow selection of one of these addressees. New 

form entries by providing different dialog boxes in which the 20 addressees which are handwritten can be added to the user's 

user can write the entries.) address book. The user can also select other options (e.g., 

The handwriting in a field is recognized by first breaking sending an image versus sending text only) in the same 
the image into segments showing characters or pieces of manner as selecting an addressee. The information which the 
characters 71. The individual image segments and combi- user enters via the graphics tablet may therefore be a mix of 
nations of these image segments are selected as character 25 handwriting and selection of particular predefined inputs. A 
candidates 72. These character candidates are assigned char- system which uses a microphone for non-textual input may 
acter classes and associated values representative of the also be configured to allow selection of predefined inputs by 
confidence with which the recognition application places the using audio prompts and corresponding menus, 
segment in the associated character class 73. The character In another embodiment, the system may be configured to 
candidates are organized into groups for which a dictionary M send faxes instead of e-mail. The foregoing description of 
look-up algorithm can be performed 74. A dictionary look- the e-mail-configured system is in large part applicable to 
up determines the word entries which best match the groups the fax-configured system. The user can enter a destination 
of character candidates 75 and may assign a level of con- fax number and/or addressee via the graphics tablet, and this 
fidence to each word. If desired, an acceptability check can information can then be transmitted to the server for con- 
be performed to determine whether the confidence level of 35 version into textual information. The textual information is 
the recognized words is sufficiently high 76. If the confi- then transmitted back to the set-top box and used by the fax 
dence level is too low, the word can be rejected and the application to dial the destination fax. As for the e-mail- 
image will be considered unrecognizable. The user can also configured system, the transmitted fax may be an image of 
be given the option of checking the textual output of the the user's handwriting, or it may be an image of textual 
recognition application and accepting or rejecting all or part 40 information corresponding to the user's handwriting. (It 
of the output 77. After the user has verified the accuracy of should be noted that in this instance, the system may convert 
the recognized text, the data can be provided to the appli- the user's handwriting to textual information so that the 
cation executing in the set-top box. It should be noted that information is more legible than handwriting or so that more 
the description is intended to be illustrative rather than information will fit on a page, but the transmitted fax will by 
restrictive and that the process of recognizing vectorized 45 its nature consist of image data rather than textual data.) 
handwriting data or voice data will differ from the foregoing Another example of an application with which the system 
description. can be employed is electronic commerce service. Electronic 

The recognition of the user's handwriting may be assisted commerce services are those through which the user can buy 

by the association of contextual information with the hand- items or otherwise conduct business. These services include 

writing. If the user's handwriting comprises entries on a 50 online catalogs and home shopping services. When using an 

form, identifying the type of information requested for each online catalog, the user can browse through the catalog to 

entry may make it easier to interpret the entry itself. For determine which products he or she would like to order. The 

example, an entry in a box requesting a social security user can provide item information via menu entries, but must 

number should contain nine digits. A character in this entry enter non-standardized information such as a shipping 

which might be interpreted as a "1", an "I" or and "1" must 55 address via the graphics tablet. 

be the numeral "1". The identification of keywords such as In another embodiment, the user can enter his information 
"to" or "cc:" may likewise distinguish the handwriting by voice. The user can use a microphone or a telephone 
which follows as a name or address. handset to provide voice data to the system. The microphone 
The system described above can be used with a number of may a special -purpose microphone for use with the interac- 
different applications. For example, an interactive television 60 tive television system or it may be a telephone handset. A 
service provider may wish to provide e-mail service to special-purpose microphone may be connected to the set-top 
subscribers. The user can select the e-mail application box, or it may be built into a remote control for the system, 
furnished by the service provider and proceed to write the A telephone handset may be connected to the set-top box, or 
message which he or she wishes to send on the graphics it may be connected directly to the return path (i.e., tele- 
tablet. In the message, the user writes the address of the 65 phone line.) The voice data is transmitted to the server, 
intended recipient and the message to be sent to the recipi- which uses voice recognition software to convert the voice 
ent. The graphical data is transmitted to the server, which data into textual data. The textual data is returned to the 
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set-top box, where it can be displayed to the user. The user 
can correct the text or confirm that the text has been 
accurately generated from the voice data. 

While the present invention has been described with 
reference to particular embodiments, it will be understood 
that the embodiments described above are illustrative and 
that the scope of the invention is not limited to these 
embodiments. Many variations, modifications, additions and 
improvements to the described embodiments are possible. 
These variations, modifications, additions and improve- 
ments are intended to be within the scope of the invention as 
detailed within the following claims. 

What is claimed is: 

1. An interactive television system comprising: 

a receiving station (20) configured to receive a broadcast 
signal, wherein said receiving station is configured to 
receive executable interactive application code corre- 
sponding to an interactive application via said broad- 
cast signal, wherein said receiving station (20) is con- 
figured to execute said interactive application, wherein 
said interactive application includes a function which 
requires input in a textual format, and wherein said 
receiving station includes an input device configured to 
receive non-textual information from a user; and 

a remote server coupled to said receiving station by a 
transmission medium, wherein said server is configured 
to receive said non- textual information from said 
receiving station and to convert said non-textual infor- 
mation into textual information, wherein said server is 
further configured to provide said textual information 
to said interactive application. 

2. The interactive television system of claim 1 further 
comprising a broadcast station coupled to said receiving 
station by a broadcast channel. 

3. The interactive television system of claim 2 wherein 
said server is coupled to said broadcast station and wherein 
said transmission medium comprises a return path between 
said broadcast station and said receiving statioo. 

4. The interactive television system of claim 3 wherein 
said receiving station includes a first modem, wherein said 
broadcast station includes a second modem and wherein said 
return path comprises a telephone line coupled to said first 
and second modems. 

5. The interactive television system of claim 3 wherein 
said return path comprises a portion of the bandwidth of said 
broadcast channel between said broadcast station and said 
receiving station. 

6. The interactive television system of claim 1 wherein 
said input device comprises a graphics tablet. 

7. The interactive television system of claim 6 wherein 
said graphics tablet comprises a digitizer configured to 
generate data corresponding to handwriting inscribed 
thereon. 

8. The interactive television system of claim 7 wherein 
said graphics tablet further comprises a display overlaid with 
said digitizer and configured to display an image of said 
handwriting as said handwriting is inscribed on said digi- 
tizer. 

9. The interactive television system of claim 1 wherein 
said receiving station further comprises a display and 
wherein said receiving station is configured to present said 
textual information on said display for verification of said 
textual information by said user. 

10. The interactive television system of claim 1 wherein 
said input device comprises a microphone. 

11. A set-top box for use in an interactive television 
system, said system having a remote server configured to 
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convert non-textual information to corresponding textual 
information, the set- top box comprising: 
receiving means (31) configured to receive executable 
interactive application code corresponding to an inter- 
active application via a broadcast signal; 
a microprocessor configured to execute said interactive 
application, wherein said interactive application 
includes a function which requires textual information 
from a user, 

an input device for receiving non -textual information 
from said user; and 

transmitting means coupled to said input device for trans- 
mitting said non-textual information to said remote 
server; and 

wherein said receiving means is configured to receive said 
corresponding textual information from said server, 
and convey said corresponding textual information to 
said microprocessor for use by said interactive appli- 
cation. 

12. The set-top box of claim 11 wherein said input device 
comprises a graphics tablet configured to receive handwrit- 
ing inscribed thereon by said user and wherein said non- 
textual information comprises digitized information corre- 
sponding to said handwriting. 

13. The set-top box of claim 11 wherein said input device 
comprises a microphone configured to receive voice infor- 
mation. 

14. The set-top box of claim 11 wherein said transmitting 
means comprises a modem. 

15. The set-top box of claim 11 wherein said receiving 
means comprises a broadcast receiver. 

16. The set-top box of claim 11 wherein said micropro- 
cessor is configured to prompt said user for said non-textual 
information, to receive said non-textual information and to 
provide said non -textual information to said transmitting 
means. 

17. A method in an interactive television system for 
providing textual input to an interactive application execut- 
ing in a set-top box of said system, the method comprising: 

receiving a broadcast signal, wherein said broadcast sig- 
nal comprises executable interactive application code 
corresponding to an interactive application, wherein 
said interactive application includes a function which 
requires textual input; 

executing said interactive application; 

providing information in a non-textual form to said inter- 
active application in said set-top box; 

conveying said information in said non-textual form from 
said set-top box to a server, 

converting said information from said non-textual form to 
a textual form in said server; 

providing said information in said textual form to said 
interactive application. 

18. The method of claim 17: 

wherein said non-textual form of said information com- 
prises images of characters; and 

wherein providing said information in said non-textual 
form comprises drawing said images on a graphics 
tablet coupled to said interactive television system. 

19. The method of claim 17: 

wherein said non-textual form of said information com- 
prises digitized image data; and 



09/15/2002, EAST Version: 1.03.0002 



US 6,345389 Bl 



13 



wherein providing said information in said non-textual 
form comprises drawing images of characters on a 
graphics tablet coupled to said interactive television 
system and digitizing said images and produce said 
digitized image data. 

20. The method of claim 17: 

wherein said non-textual form of said information com- 
prises images of characters; and 

wherein converting said information from said non- 
textual form to said textual form comprises executing a 
handwriting recognition application on said server and 
providing said images to said application to produce 
said textual form of said information. 
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21. The method of claim 17: 

wherein said non-textual form of said information com- 
prises voice data; and 

wherein providing said information in said non-textual 
form comprises speaking into a microphone coupled to 
said interactive television system and said microphone 
producing said voice data from spoken words. 

22. The method of claim 17 wherein converting said 
information from said non-textual form to a textual form in 
said server comprises converting said non-textual form of 
said information to ASCII characters. 
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